[branch-55] Reduce datafusion-spark dependency footprint (#24351) - #25277
Merged
Merged
Conversation
## Which issue does this PR close? N/A ## Rationale for this change `datafusion-spark` pulls unnecessary crates into its production dependency graph, increasing compilation time and artifact size for downstream users. ## What changes are included in this PR? - Replace compatibility re-exports with the narrower crates that define `TableFunction` and `FunctionRegistry`. - Remove the unused `crypto_expressions` activation; Spark provides its own SHA-1, SHA-2, and CRC32 implementations. - Fix the `quote` import so builds without the optional `core` feature continue to compile. These changes reduce the production dependency graph from 279 to 257 packages. Cargo Machete does not report these dependencies because they are referenced in source through re-exports or activated through Cargo features. ## Are these changes tested? Yes ## Are there any user-facing changes? No. Public APIs and default behavior remain unchanged.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## branch-55 #25277 +/- ##
=============================================
- Coverage 81.21% 81.21% -0.01%
=============================================
Files 1110 1110
Lines 388499 388499
Branches 388499 388499
=============================================
- Hits 315517 315513 -4
- Misses 54432 54433 +1
- Partials 18550 18553 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
AdamGS
approved these changes
Sep 18, 2026
martin-g
approved these changes
Sep 18, 2026
Contributor
Author
|
The CI failure is due to the minio images being out of date. I made a PR to backport them as well to 55: |
alamb
added a commit
that referenced
this pull request
Sep 19, 2026
…25485) - Fixes #25215 on branch-55 This PR: - Backports #25092 and #25216 from @comphead to the branch-55 line ## Rationale MinIO withdrew `minio/minio` from Docker Hub on 2026-09-11, so every `datafusion-cli` storage integration test on `branch-55` now panics while pulling the image (for example the CI failure on #25277). --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com>
Contributor
Author
|
and here is a PR to fix the security audit failure 🙄 |
alamb
added a commit
that referenced
this pull request
Sep 19, 2026
…25514) - Fixes the `security_audit` failure on `branch-55` (for example https://github.com/apache/datafusion/actions/runs/35435408307/job/105877099456 on #25277) This PR: - Backports #25309 from @Xuanwo to the branch-55 line ## Rationale `branch-55` locks `rustls` 0.23.39, which is affected by [RUSTSEC-2026-0285](https://rustsec.org/advisories/RUSTSEC-2026-0285.html) ("TLS 1.3 handshake messages incorrectly accepted across encryption level boundaries", published 2026-09-14). Every PR targeting `branch-55` now fails the `security_audit` job until the lockfile is updated. `rustls` 0.23.45 fixes the vulnerability. The cherry-pick applied cleanly: `branch-55` and `main` had identical `rustls`, `aws-lc-rs`, `aws-lc-sys`, and `rustls-webpki` entries before the fix. ## Testing `cargo audit` with the same ignores as the CI job passes on this branch (it fails on `branch-55` without this change): ```sh cargo audit --ignore RUSTSEC-2026-0194 --ignore RUSTSEC-2026-0195 ``` `cargo fetch --locked` and `cargo check --locked -p datafusion-cli` also succeed, confirming the updated lockfile resolves and builds. Co-authored-by: Xuanwo <github@xuanwo.io>
Contributor
Author
|
The CI finally passes! |
Contributor
Author
|
Thank you everyone |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
55.1.0(minor/patch) Release (Sep 2026) #24462datafusionin this scope" #24474 on branch-55This PR:
Rationale
As noted in #24462 (comment), publishing
datafusion-sparkfor the 55.1.0 release required a manual local change because the crate does not compile without its optionalcorefeature (see #24474). Backporting this fix means a future 55.2.0 release can be published without manual intervention.Verified locally that
cargo check -p datafusion-spark --no-default-featuresnow succeeds on this branch (it fails onbranch-55without this change).